Skip to content

Orchestrator: multi-repo polling for asm_* module CI - #400

Merged
andrewjong merged 2 commits into
developfrom
orchestrator-multi-repo
Aug 29, 2026
Merged

Orchestrator: multi-repo polling for asm_* module CI#400
andrewjong merged 2 commits into
developfrom
orchestrator-multi-repo

Conversation

@andrewjong

Copy link
Copy Markdown
Member

One orchestrator instance now serves trunk and the module repos: config.yaml takes a repos: list (legacy repo: still accepted), the spawn loop collects queued airstack-ephemeral jobs across all of them with per-repo JIT runner minting, and reaping/orphan-sweep/max_concurrent span the set. Unblocks GPU CI for asm_dfm2_disturbances / asm_optitrack / asm_macvo (their ci.yml callers now pin the reusable workflow @develop).

Host-side rollout (orchestrator machine): pull this commit, switch /etc/airstack-orchestrator/config.yaml to the repos: list, extend the PAT to the three module repos (Actions + Administration read/write), systemctl restart airstack-orchestrator, and confirm the startup line shows repos=[...].

Unit suite: 400 passed / 7 skipped. VERSION → 0.20.0-alpha.10; Release Notes updated.

🤖 Generated with Claude Code

…module repos

The spawn loop gathers queued jobs across every configured repo (JIT
runner configs minted per-repo), state entries record their repo for
reaping, and the orphan sweep / max_concurrent cap span the whole set.
The legacy singular repo: key is still accepted; pre-multi-repo state
entries reap against the first listed repo. config.example.yaml lists
trunk plus the three asm_* module repos; README's run-a-second-instance
procedure is replaced by the repos: list.

Bump version to 0.20.0-alpha.10.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Test Metrics — 71456dab39aab10b5d3d8d2e3af73beee83ce5be

No metrics report generated.

@github-actions

Copy link
Copy Markdown
Contributor

Test Metrics — 0404a0be4de43b335a08fc76c54600d2de0138a1

This was a unit/build-only run. Simulation regression comparison does not apply.

system.test_build_packages

Pass rates

Test Pass Fail Skip Rate
test_colcon_build_gcs 1 0 0 100%
test_colcon_build_ms_airsim 1 0 0 100%
test_colcon_build_robot 1 0 0 100%
test_colcon_test_robot 1 0 0 100%

Metrics

Test Metric Value
test_colcon_build_robot duration_s 87.64s
test_colcon_test_robot duration_s 39.6s
test_colcon_build_gcs duration_s 68.31s
test_colcon_build_ms_airsim duration_s 15.11s

@andrewjong
andrewjong merged commit c5fc15c into develop Aug 29, 2026
5 checks passed
andrewjong added a commit that referenced this pull request Aug 29, 2026
…ence docs

Scope cut agreed with Andrew before merge:

- Drop tests/system/test_optitrack_e2e.py: it sets LAUNCH_NATNET (a removed
  no-op shim) and targets robot/ros_ws/src/perception/natnet_ros2, which was
  extracted from trunk (70423c4). It could never pass on develop and PR CI
  never executes the optitrack mark, so it would rot silently. The same test
  lives where it belongs: asm_optitrack/tests/system/, exercised by the
  module CI that is now served by the multi-repo orchestrator (#400).
  The mark stays registered in tests/pytest.ini for module-CI runs.
- Drop tests/report-requirements.txt: develop's metrics job already installs
  tests/requirements.txt (#407); a second requirements file for the same
  tree is a drift hazard. The workflow contract now pins the #407 behavior.
- Drop harness/image_prep.py and the image-preparation.json workflow
  instrumentation: metadata plumbing whose value doesn't carry its weight —
  the associative-array bash was the most fragile code in the PR. The
  image-prep step returns to develop's plain version.
- diagnostics: drop LAUNCH_NATNET from SAFE_ENV_KEYS (dead); keep
  PX4_PARAM_SET (live compose env-file selector).
- Un-clobber develop content the branch predated: run-system-tests SKILL
  description (waypoint_flight, wiring marks), BSD-3-Clause-Clear license,
  `airstack images build` spelling.
- Reference docs refreshed to where CI stands now: AGENTS.md test-suite
  paragraph (schema-v2 run_meta, advisory metric deltas, diagnostics
  bundle, fingerprint-only comparisons) and --config-only intent flag;
  module_ci.md + module-system-tests.yml header (orchestrator polls a
  repos: list — asm_optitrack included today); Release Notes entry for the
  trustworthy-outcomes policy.

Full unit suite on this branch: 419 passed, 0 failed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
andrewjong added a commit that referenced this pull request Aug 29, 2026
* Make system-test outcomes trustworthy and actionable

Separate infrastructure failures, test assertions, and advisory metric changes so CI blocks only on real test or integrity failures while preserving comparable performance evidence.

* Classify readiness failures as infrastructure

Preserve explicit infrastructure intent through pytest call reports so simulator startup crashes cannot be misreported as algorithm assertions.

* Allow focused manual flight campaigns

Expose trajectory and takeoff sweeps in workflow dispatch so CI validation can run minimal algorithm samples before expanding to expensive matrices.

* Finish leftover merge markers from the develop rebase.

Keep develop's images CLI and Using CI layout while preserving this PR's advisory metrics, diagnostics artifacts, and infrastructure mark.

* Fix unit-test merge fallout and bump VERSION.

Keep --config-only hermetic for fleet contracts, still fail AUTONOMY_ROLE as a config check, and document the new help tokens so CI can pass the increment gate.

* Strip post-extraction residue and metadata plumbing; refresh CI reference docs

Scope cut agreed with Andrew before merge:

- Drop tests/system/test_optitrack_e2e.py: it sets LAUNCH_NATNET (a removed
  no-op shim) and targets robot/ros_ws/src/perception/natnet_ros2, which was
  extracted from trunk (70423c4). It could never pass on develop and PR CI
  never executes the optitrack mark, so it would rot silently. The same test
  lives where it belongs: asm_optitrack/tests/system/, exercised by the
  module CI that is now served by the multi-repo orchestrator (#400).
  The mark stays registered in tests/pytest.ini for module-CI runs.
- Drop tests/report-requirements.txt: develop's metrics job already installs
  tests/requirements.txt (#407); a second requirements file for the same
  tree is a drift hazard. The workflow contract now pins the #407 behavior.
- Drop harness/image_prep.py and the image-preparation.json workflow
  instrumentation: metadata plumbing whose value doesn't carry its weight —
  the associative-array bash was the most fragile code in the PR. The
  image-prep step returns to develop's plain version.
- diagnostics: drop LAUNCH_NATNET from SAFE_ENV_KEYS (dead); keep
  PX4_PARAM_SET (live compose env-file selector).
- Un-clobber develop content the branch predated: run-system-tests SKILL
  description (waypoint_flight, wiring marks), BSD-3-Clause-Clear license,
  `airstack images build` spelling.
- Reference docs refreshed to where CI stands now: AGENTS.md test-suite
  paragraph (schema-v2 run_meta, advisory metric deltas, diagnostics
  bundle, fingerprint-only comparisons) and --config-only intent flag;
  module_ci.md + module-system-tests.yml header (orchestrator polls a
  repos: list — asm_optitrack included today); Release Notes entry for the
  trustworthy-outcomes policy.

Full unit suite on this branch: 419 passed, 0 failed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Pranav Kumara <pkumara@andrew.cmu.edu>
Co-authored-by: Andrew Jong <ajong@andrew.cmu.edu>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant